Skip to content

Conversation

@ochafik
Copy link
Contributor

@ochafik ochafik commented Jan 18, 2025

No description provided.

@ochafik ochafik merged commit 297911f into main Jan 18, 2025
7 checks passed
@ochafik ochafik deleted the hf-token branch January 18, 2025 17:34
ochafik referenced this pull request in ochafik/minja Nov 2, 2025
Authenticate CI w/ read-only HF Token
aldehir added a commit to aldehir/minja that referenced this pull request Jan 1, 2026
…gle#21)

Fixes google#16

Looks like `std::regex_replace()` does not respect anchors, at least not
in Windows.

**Minimal reproducing example (Microsoft (R) C/C++ Optimizing Compiler
Version 19.44.35221 for x64)**
```cpp
#include <iostream>
#include <regex>

int main() {
    auto text = "\nthis contains\n\nmultiple\nline\n\nbreaks\n\n";

    std::cout << "== Leading ==\n";
    auto bad = std::regex_replace(text, std::regex(R"(^\s)"), "");
    std::cout << "Bad: " << bad << "\n";
    std::cout << "==\n";

    std::string good = text;
    good.erase(0, good.find_first_not_of(" \t\r\n"));
    std::cout << "Good: " << good << "\n";
    std::cout << "==\n";

    std::cout << "== Trailing ==\n";
    bad = std::regex_replace(text, std::regex(R"(\s$)"), "");
    std::cout << "Bad: " << bad << "\n";
    std::cout << "==\n";

    good = text;
    auto pos = good.find_last_not_of(" \t\n\r\f\v");
    good.resize(pos == std::string::npos ? 0 : pos + 1);
    std::cout << "Good: " << good << "\n";
    std::cout << "==\n";
}
```
```
== Leading ==
Bad: this contains
multiple
line
breaks

==
Good: this contains

multiple
line

breaks


==
== Trailing ==
Bad: 
this contains
multiple
line
breaks
==
Good: 
this contains

multiple
line

breaks
==
```

Passes all the tests, excluding the gated templates I don't have.

```
$ ctest -R test-supported-template -j 24
...
100% tests passed, 0 tests failed out of 220

Total Test time (real) =  32.38 sec

The following tests did not run:
         11 - test-supported-template-google-gemma-7b-it (Skipped)
         12 - test-supported-template-CohereForAI-c4ai-command-r-plus (Skipped)
         14 - test-supported-template-meta-llama-Llama-3.2-3B-Instruct (Skipped)
         15 - test-supported-template-meta-llama-Llama-3.1-8B-Instruct (Skipped)
         16 - test-supported-template-meta-llama-Meta-Llama-3-8B-Instruct (Skipped)
         18 - test-supported-template-meta-llama-Llama-2-7b-chat-hf (Skipped)
         54 - test-supported-template-CohereForAI-aya-expanse-8b (Skipped)
         55 - test-supported-template-databricks-dbrx-instruct (Skipped)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant